-
-
Notifications
You must be signed in to change notification settings - Fork 2.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Do not fail to build if 'zig build-lib' etc. arguments exceed OS limits #10950
Conversation
This change refactors the `zig` argument handling (for `build-lib`, etc. commands) to use a `process.argsWithAllocator` iterator instead of directly accessing arguments via array indices. This supports the next commit which will enable us to use a response file argument iterator here seamlessly. Helps #10693 Signed-off-by: Stephen Gutekanst <stephen@hexops.com>
This change enables `zig build-lib` and friends to take a response file of command line arguments, for example: ```sh zig build-lib @args.rsp ``` Which effectively does the same thing as this in Bash: ```sh zig build-lib $(cat args.rsp) ``` Being able to use a file for arguments is important as one can quickly exceed the 32 KiB limit that Windows imposes on arguments to a process. Helps #10693 Signed-off-by: Stephen Gutekanst <stephen@hexops.com>
In Mach engine we're seeing command line arguments to `zig build-lib` exceed the 32 KiB limit that Windows imposes, due to the number of sources and compiler flags we must pass in order to build gpu-dawn. This change fixes the issue by having `Builder` check if the arguments to a `zig build-*` command are >30 KiB and, if so, writes the arguments to a file `zig-cache/args/<SHA2 of args>`. Then the command invocation merely becomes `zig build-lib @<that file>`. Fixes #10693 Fixes hexops/mach#167 Signed-off-by: Stephen Gutekanst <stephen@hexops.com>
I've fixed the issue in Zig upstream: ziglang/zig#10950 Helps #86 Signed-off-by: Stephen Gutekanst <stephen@hexops.com>
I've fixed the issue in Zig upstream: ziglang/zig#10950 Helps #86 Signed-off-by: Stephen Gutekanst <stephen@hexops.com>
I've fixed the issue in Zig upstream: ziglang/zig#10950 Helps hexops/mach#86 Signed-off-by: Stephen Gutekanst <stephen@hexops.com>
|
I was not able to find such an API, maybe it'd be something undocumented? In any case, I should note that macOS and Linux also have limitations here (256 KiB and 256 MiB, respectively): I think it would be advisable to have response file support like this in place, as clang and others use the same approach to solve this issue. |
Signed-off-by: Stephen Gutekanst <stephen@hexops.com>
Signed-off-by: Stephen Gutekanst <stephen@hexops.com>
Signed-off-by: Stephen Gutekanst <stephen@hexops.com>
Signed-off-by: Stephen Gutekanst <stephen@hexops.com>
Addressed all the feedback :) |
Signed-off-by: Stephen Gutekanst <stephen@hexops.com>
friendly ping :) I have some quite cool tech demos that will be blocked on this fix landing |
Thanks for the enhancement! Looks like this introduce a regression in
|
Reverted in 5ab5e2e |
These patches fix a major issue which prevents using Zig's build system from being viable in larger projects with lots of source files on Windows (and macOS + Linux, to a lesser degree.)
In Mach engine we're seeing command line arguments to
zig build-lib
exceed the 32 KiB limit that Windows imposes in theCreateProcess
API, due to the number of source files and compiler flags we need in order to build WebGPU/Dawn.I've fixed this issue in three steps (best reviewed as individual commits):
zig build-lib [args]
args handling to use an iterator, in anticipation of response file support.zig build-lib @args.rsp
(andbuild-exe
, etc. friends) so you can pass them a text file of arguments instead.Builder
to check if the arguments it would pass to anyzig
build command are >30 KiB and, if so, use e.g. azig build-lib @zig-cache/args/<SHA2 of args>
response file instead.Fixes #10693
Fixes hexops/mach#167